System and methods for computerized health and safety assessments

ABSTRACT

Systems and methods are provided for measuring, assessing, predicting, improving and presenting the state of physical object and human core temperatures, using imaging devices, e.g., a thermal infrared camera, and/or intruders in a region of interest to an operator, such that little or no operator effort is required to install, use or receive reports from the system. The invention also includes, for example, means and methods for exploiting autonomous operation and configuration, placement at remote sites, enhancement of image resolution and estimation of range such that accuracy of results and autonomy of operation is enhanced.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 63/013,081, filed Apr. 21, 2020, and is a Continuation in Part ofU.S. patent application Ser. No. 16/779,622, filed Feb. 2, 2020, theentire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present invention generally relates to sensor data collection andprocessing for safety and operational effectiveness. More particularly,the present invention relates to thermal and/or color image sensor datacollection and image processing for the purpose of industrial siteeffectiveness, e.g., preventative and predictive maintenance, andsafety, e.g., thermographic measurement of industrial assets in spaceand time through automated infrared thermographic imaging andinterpretation.

BACKGROUND

In the field of industrial thermography, the current standard practiceis to use handheld devices to make image measurements and subsequentlycombine these image measurements with additional supporting physicalmeasurements (e.g., atmospheric conditions for absorption). Sequences ofmanual operations are often combined with computer assisted operationsto produce reports corresponding to the point in time at which thehandheld measurements were made.

Thus, present day practices for the measurement of the physicaltemperature of industrial equipment (for example, breakers, fuses,switches, and other circuit protection devices and components housedwithin a switchgear cabinet) often involve direct measurement using aninstrument in contact with a region of the equipment, e.g., a bus barconnection fastener, and/or human measurement using a handheldthermographic device or the like, and may also include manualmeasurements and assessments of contributing factors. Contributingfactors for handheld thermographic devices may include, for example,equipment optical properties, environmental properties, and sources ofthermal energy other than the equipment being assessed. Such additionalassessments are made to increase the accuracy of the equipmenttemperature reported by the handheld thermographic device.

Measuring with an instrument in contact with the equipment, e.g., aresistance temperature detector (RTD), thermistor or thermocouple, oftenappears to be the least ambiguous method for measuring physicaltemperature. However, such an instrument measures only a single point ofan object and does not provide information about the context of themeasurement such that assessments of heat relative to a context could bemade. Also, directly contacting an energetic surface, i.e., a highlyenergized electrical connection, can introduce risk to the instrumentand the equipment. Further, in the event of an instrument failure,replacement can be cost prohibitive when de-energizing critical(continuously operating) equipment is required to do so.

Such manual measurements can be valuable to the owners and operators ofequipment, but often the equipment being assessed is in a dangerousarea, e.g., highly energized electrical switch gear, or in a dangerousstate, such as on the verge of overheating and igniting due to a loosebus bar connection. In a dangerous area, e.g., inside a cabinet housingelectrical switch gear, safety protocols often prohibit making a manualmeasurement without first de-energizing the equipment. Since theequipment often is vital to some valuable process that requirescontinuous equipment operation, de-energizing is ill-advised foreconomic reasons. Further, even if there are occasional opportune timesto de-energize and make a measurement, since underlying thermalprocesses for the measured equipment typically vary on a scale ofminutes or hours, producing a single measurement on a yearly or even amonthly scale can lead to erroneous or misleading indicators of healthand status.

At the same time, there is also a known risk of unintended intrusion,e.g., so-called “critter events,” at some equipment sites that endangerindustrial assets. Consequently, it is advantageous to use bothintrusion detection and thermography functions so as to minimize injuryto equipment or humans who use or visit the equipment.

The present invention addresses these and other limitations of the priorart.

SUMMARY OF THE INVENTION

The following is a summary of the invention intended to provide a basicunderstanding of some aspects of the invention. This summary is notintended to identify key or critical elements of the invention or todelineate the scope of the invention. Its sole purpose is to presentvarious concepts of the invention in a simplified form as a prelude tothe more detailed description and the defining claims that are presentedlater.

The present invention relates to systems and methods for measuring,assessing, predicting, improving, and presenting the state of physicalobject temperatures using imaging devices, e.g., a thermal infraredcamera, and/or biological organisms, e.g., intruders or subjects, in aregion of interest to an operator, such that little or no operatoreffort is required to use or receive reports from the system. Variousembodiments of the invention are particularly useful for generatingintuitive, real-time composite thermal images of heat-generatingcomponents within an enclosure, such as a switchgear cabinet or otherenclosed space where thermal monitoring is inconvenient and/ordangerous.

These and other features and advantages of the invention will beapparent to those skilled in the art from the following detaileddescription of preferred embodiments, taken together with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an embodiment of the invention;

FIG. 2 is a block diagram illustrating an embodiment of using theinvention for automated thermography and intrusion detection;

FIG. 3 is a block diagram illustrating an embodiment of using theinvention for generating an object catalogue;

FIG. 4 is a block diagram illustrating an embodiment of the inventionthat includes a calibration target for enhanced thermographicperformance;

FIG. 5 is a block diagram illustrating an embodiment of the inventionthat includes a thermographic sensor for equipment monitoring,thermographic sensors for human monitoring, and a calibration target forenhancing performance of thermographic systems, all as part of a largerenterprise control system;

FIG. 6 is an illustration of an embodiment of the invention that usestwo or more thermographic sensors with diverse fields of view in orderto produce elevated accuracy for sensors having overlapping fields ofview;

FIG. 7 is an exemplary thermal image useful in describing variousaspects of the present invention;

FIG. 8 is a graph of absolute temperature difference associated with aportion of the objects illustrated in FIG. 7 ;

FIG. 9 illustrates the monitoring of components housed within anenclosure in accordance with an example embodiment;

FIG. 10 illustrates a sensor module in accordance with one embodiment;

FIG. 11 illustrates a compact sensor module in accordance with oneembodiment in which the module is collinear with the attached cabling,thereby minimizing electric field gradients; and

FIG. 12 illustrates a composite thermal image in accordance with oneembodiment.

DETAILED DESCRIPTION OF PREFERRED Exemplary Embodiments

In general, the present invention relates to the automation ofindustrial thermography. In that regard, the following detaileddescription is merely exemplary in nature and is not intended to limitthe inventions or the application and uses of the inventions describedherein. Furthermore, there is no intention to be bound by any theorypresented in the preceding background or the following detaileddescription. In the interest of brevity, conventional techniques andcomponents related to thermal imaging, image processing, computerprocessors, and calibration methods may not be described in detailherein as such topics are well known by those of ordinary skill in theart.

In accordance with one embodiment, in order to avoid unnecessary humanrisk resulting from measurement and/or unwanted intrusion, e.g., byliving organisms such as animals and humans, the present inventionenables the automation of thermographic measurement and intrusiondetection such that a single system mitigates risk of harm to equipmentand humans in an enterprise. Toward that end, embodiments of the presentinvention relate to an autonomous industrial security and safety systemincluding one or more imaging devices with integral computing and datastorage capacities configured in a network to which additional computersand storage devices may be connected, and to which a user may connect inorder to access raw and processed data, and from which a user mayreceive automated communications concerning the current and likelyfuture state of the physical assets being monitored. In accordance withone embodiment, the imaging device(s) comprise a modular multispectralimaging system for which a multiplicity of image sensor modules are usedto measure and assess physical temperature across one or more distinctregions of interest throughout an industrial asset of interest, e.g., asequence of switch gear cabinets each containing equipment and points ofconnection that are to be monitored.

In accordance with various embodiments, at least one non-visual (e.g.,thermal infrared) image device is used for imaging objects in its systemfield of view, so that the thermal emission of imaged objects can beused to estimate object physical temperature, including proximate humanworkers, from the measured thermal infrared radiance imaged by thethermal camera. Calibration function, e.g., for improving estimates ofphysical temperature when required beyond that included with thermalcamera modules, is supported with modules having integral thermographiccalibration features and that can be networked with devices used forimaging. Intrusion function, e.g., perimeter security, is enabled byusing thermal or other sensor or camera data to detect and report thepresence of biological organism activity in areas for which suchactivity is of concern, e.g., closed-access regions in proximity to theequipment. Computers that are integral to system imaging devices, e.g.,onsite or embedded computers, may produce some or all of the dataproducts required to achieve the security and safety functions.

Computing devices (e.g., desktop computers, laptop computers, tabletcomputers, smartphones, or the like) that are connected to systemimaging devices and sensor modules by way of a network connection mayalso produce data products and will often be used for both theproduction of data products and associated reports, graphs, alerts andother results of interest to owners or operators of the industrialequipment being monitored. Intrusion detection function extends tocoordinating with separate cueing or cue-able devices located remotefrom the invention that respond to cueing events, e.g., transmitted bysystems in accordance with the invention, or that independently detectevents and transmit signals to the invention that it interprets and usesto adjust its monitoring capabilities, e.g., by recording continuously,and/or by adapting detection standards or criteria through so-calledtraining of algorithms.

Systems in accordance with the invention may be configured for equipmentand intrusion monitoring manually by an operator, e.g., by sitting nearthe invention or remotely at a desk over a network, or via automaticconfiguration. Automatic configuration of the system may involve, forexample: using one or more computers and including thermal and/or otherspectral imaging devices, e.g., one that is sensitive to visible light,to algorithmically detect and identify relevant objects or spaces,classify those objects and spaces, estimate their physical propertiesand the physical properties of their environment (e.g., such as wouldcontribute to and affect an interpretation of temperature based onmeasurements, including but not limited to thermal radiance of nearbyobjects, atmospheric loss and scattering as a function of path length,optical path length, etc.), and initialize data collection databaseswith these inferred object properties using local storage and/or remote(e.g., cloud) storage.

Automation enabled by embodiments of the invention also includes the useincluded or nearby imaging and sensing devices to localize equipment orspaces, e.g., estimating distances from imaging devices to equipment orspaces in the measurement area, and combining that data with GPS orother locating techniques to determine the sensor module's position on amap, and the production of enhanced resolution (or “super-resolution)using predetermined or measured properties of the imaging devices andrelated optics (e.g., point spread function), or scene knowledgegathered with included or nearby imaging devices, e.g., a color cameramodule. The invention also addresses the automation of the estimation ofcalibration parameters and equipment physical parameters, e.g.,emissivity, using physical observables in the environment, for example,atmosphere/sky, stellar objects, identifiable solids, and historicaldata on such objects and materials, such historical data providing timevarying observables against which one may estimate unknowns that thenpermit accurate assessment of observables through appropriate dataanalytics and/or machine learning techniques. The present inventioncontemplates achieving such automation through the use of data alone orin combination with installed calibration objects and/or physical andmathematical models of underlying phenomena.

With reference to the conceptual block diagram of FIG. 1 and inaccordance with one embodiment of the invention, a site may be rapidlyand autonomously monitored for both thermography and intrusion using asystem as shown. In general, one or more sensor modules (100, 140-142)interface with a base station 110 and one or more networked computingsystems 120 and 130. Sensor module 100, in the illustrated embodiment,includes a visible/non-thermal imager 102, a thermal infrared camera (orsimply IR camera) 103, an auxiliary sensor 109, a local computerprocessor 101, a light source 108, power conditioning system 104,environmental orientation and/or locationing module 105, device bus ornetwork interface 106, and local data storage system 107. Sensor modules140-142 may also include one or more of the components illustratedwithin sensor module 100.

Base station 110 includes local data storage 111, radio 115, localcomputer processor 112, local sensors 116, network bus or networkinterface 113, and LAN/WAN network interface 114. Networked computingsystem 120 includes a remote server 121, a LAN/WAN network interface122, and remote data storage 123. Similarly, networked computing system130 includes remote server 131, LAN/WAN network interface 132, andremote data storage 133.

While the system of FIG. 1 may be implemented in a variety ofenvironments and contexts, one particular use of the present inventioninvolves monitoring the thermal characteristics of heat-generatingobjects with an enclosed space, such as a switch-gear cabinet of thetype used to house high-voltage components. Referring to FIG. 9 , forexample, an enclosure 901 (e.g., a switchgear cabinet) is used to housean array of switchgear components (generally, “objects”) such as objects902 (e.g., objects 902A, 902B, and 902C). Enclosure 901 will typicallyinclude an access door or other access feature 930. The internal objects902 generally include power-conducting components (such as switches,bus-bars, circuit breakers, fuses, lightning arrestors, and the like)and control systems (such as control panels, current transformers,relays, etc.). It will be appreciated that the invention is not solimited, however, and that these components are only provided asexamples.

With continued reference to FIG. 9 , one or more sensor modules 910(e.g., 910A-910C, which may correspond to items 100 and 140-142 in FIG.1 ) are suitably mounted within enclosure 901 so that they can monitorthe thermal characteristics and/or other attributes of objects 902. Inthe illustrated embodiment, for example, three sensor modules 910A,910B, and 910C are shown mounted adjacent to respective objects 902A,902B, and 902C. Sensor modules 910 are oriented such that their sensorshave respective fields of view (e.g., field of view 911 for sensormodule 910C) including a surface of the objects being monitored.

As shown, sensor modules 910 may be interconnected by a set of cables(such as cable 906), which preferably provide both data communication aswell as power for the sensor modules 910 (e.g., a CAN bus+power cable).As illustrated in FIG. 9 , and as described in further detail below,sensor modules 910 may be configured such that they have multipleinput/output connectors and can thus be chained together (e.g., in aloop, in series, or “daisy chained”), thereby reducing cablingrequirements and resulting in a much more compact system than providedby the prior art.

The sensor modules 910 are communicatively coupled to a base unit 920(e.g., item 110 in FIG. 1 ), which may be mounted internal to orexternal to enclosure 901. As shown, base unit 920 is preferably coupledto a network 925, as described in further detail herein.

FIG. 10 is an exterior view of a sensor module 910 in accordance withone embodiment. In particular, sensor module 910 includes an enclosure1050 and a pair of interconnects (1011 and 1012) configured to acceptrespective cables 1001 and 1002. Enclosure 1050 includes, in thisembodiment, three circular windows providing viewports for an infraredthermal sensor 1031, an optical camera 1032 (e.g., a visible lightcamera or other device that produces images interpreted by humans asbeing equivalent or similar to photographic images formed with lightfrom the human visual spectrum, including visible light, near infrared,short wave infrared or ultraviolet wavebands), and an illuminator (e.g.,LED) 1033. It will be appreciated that the positions of these componentsis not intended to be limiting, and that the sensors and illuminator maybe positioned in a variety of ways. In a preferred embodiment, however,it is advantageous for all three windows to face in substantially thesame direction, as shown. Also shown in FIG. 10 is a pressureequalization component 1040 configured to ensure that the pressureinternal to and external to enclosure 1050 is substantially equal.

FIG. 11 illustrates yet another embodiment in which the sensor module1100 is compact and collinear with the association cabling. Moreparticularly, sensor module 1100 is has a generally cylindricalenclosure 1102 and has a major axis collinear with the axes ofinterconnects 1011 and 1012. In this way, when cables are attached toeither one or both of interconnects 1011 and 1012, enclosure 1102 itselfis collinear with the cables. As in the previous embodiment, sensormodule 1100 includes a thermal sensor 1031, an illuminator 1033, and anoptical camera 1032. In some embodiments, the diameter D of enclosure1102 is substantially equal to (or no more than 10% greater than) thediameter of the cables themselves. The result is a very compact sensormodule that, due to its orientation, also minimizes electric fieldgradients in and around sensor module 1100.

FIG. 12 illustrates an example composite image 1200 that may result fromthe use of sensor modules in accordance with the present invention. Thatis, as shown, four separate images (1201, 1202, 1203, and 1204) ofobjects being monitored are “stitched” together into one image for easyreview by an operator. In these images, temperature is indicated bygreyscale intensity; however, it will be understood that such images maybe rendered in color. Note that the individual images 1201-1204 may bepositioned so that they correspond spatially to the objects beingobserved, or may be discrete images of objects at different locationswithin an enclosure.

Referring to FIG. 1 in conjunction with the flow chart of FIG. 2 , anexemplary flow 200 of an embodiment of the invention will now bedescribed. The description that follows applies to the embodiments shownin FIG. 9-12 above as well as other embodiments encompassed by thepresent invention.

As shown, the method begins (at step 201) with the placing of a system(such as that shown in FIG. 1 ) at a known geographic location. Inaccordance with one embodiment, this placement is made straightforwardby virtue of its simplicity and portability and having autonomous meansof supplying its power, e.g., solar, wind, electric field, magneticfield, thermal (excess heat), vibrational (mechanical) or equivalentautonomous (or self-contained) or energy-harvesting power source, sothat placement constitutes an installation once a user enables power,e.g., with a mechanical or electronic (e.g., software enabled) powerswitch, for the system (step 202).

The system then finds a network (step 203), e.g., a wireless LTE or WiFimesh or a physically connected network, and connects (step 204) to aremote server 121 preconfigured for use with the system. Subsequently,the system will send GPS information, e.g., from a GPS auxiliary sensoror from a GPS integrated into the radio 115, e.g., LTE radio etc., and aunique system identifier (step 205) preconfigured at the time of systemmanufacture to the remote server 121. The system can then receive itsinitial cue-able device and proximate module device data along withrelevant and security settings (step 206). Cue-able devices are devices,e.g., a nearby imaging system with a slewing capacity, e.g.,electromechanical pan and tilt, field of view that can respond to analert from a cueing device, e.g., a module 100. Proximate devices areother modules that are similar to, the same as or complementary to,e.g., if the module is a thermographic calibration source, the module100 that can be configured or used to operate jointly with a givenmodule 100 such that, operating as a system, a plurality of modules mayenable observation and assessment of a much larger area than is possiblewith a single module, each module having capacity to accommodate thermalinfrared camera 103, supported by a visible or non-thermal imager 102,and having the option to include one or more auxiliary sensors 109 thatsupport particular modalities beyond that of a sole thermal infraredcamera 103, e.g., including but not limited to multi-spectral sensing,short wave infrared sensors, mid wave infrared sensors, ranging sensors,optical polarimetric sensors, electric field, magnetic field, or radiofrequency (“RF”) sensors. Once cue-able device and proximate modules areaccounted for, a catalogue of such devices can be built (207), e.g., asa database within local storage 107 or remote storage components 123and/or 133. The orientation of each module can then be obtained (208)using its built in orientation sensors 105 to establish an optical lineof sight for the module; this can be augmented by illuminating the scene(if light is required) with structured or unstructured illumination 108such that a non-thermal imaging device 102, e.g., color camera, can beused to collect scene data for detecting, localizing and cataloguingobjects imaged with each module 100 (209).

Using GIS or spatial data 105 obtained for or with a module both aphysical map of area coverage (210) and a logical map of possibledetection outputs (211), e.g., data products or results from object orscene state assessments/computations, such maps enable theinterpretation of module data as part of a larger system, such that, forinstance, a small animal traversing a network of cabinets containingmodules could be tracked by virtue of a logical sequence of detectionswithin a network (virtual or otherwise) of modules having sensors thatcan be interpreted for living organism detection. The logical andphysical maps thus generated can be further used to construct a graph orother structured description of plausible detection relationships (212)such that causal events can be detected and asserted. Once the necessarycalculations have been performed for the maps the maps and detectionrelationships can be stored (213) on local or remote resources.

Before system can become operational, the system can be operated inconjunction with a video management system (VMS) in which case localdevice image streams, e.g., motion JPEG, MPEG, raw, or other temporalsequences of data having one or more spatio-temporal dimensions, can beconnected to local or data recipients, a common one being a VMS. A VMSor equivalent function permits thermal and/or color video to be streamedto remote server 131 or local computer 112 such that a user might reviewprior video data or view live video data.

After storing the object catalogue (213) and connecting to image streams(214), the system begins operation (215). Subsequently, thermal sensordata is collected (217), non-thermal sensor data (such as optical data,environmental data, motion data, etc.) is collected (218). The data isthen calibrated to the scene and environment (219), and the temperaturesof the objects being monitored are assessed (220). Intrusion status mayalso be assed (221). Finally, remote and/or local databases are updated,and any alerts, reports, and messages are transmitted as appropriate(222). The process then continues, looping back to step 217 as shown.

In all of the computer-automated steps of FIG. 2 , it is understood thata given step or function can be performed manually if and when suchoperation is either the only available option for a particular situationor if manual operation is a preferred mode for reasons peculiar to agiven end user. It is also understood that, given an object catalog fora site installation, the objects can be selected and prioritized forthermography or selected and prioritized for intrusion, such that endusers are able to organize explicitly for intrusion related actions orthermography related actions, or a blend of the two. These prioritysettings are either provided through dialogue with end users, e.g., acustomer service phone call prior to installation after which time acustomer service agent enters the data into the user's profile for thesite etc., or the priority settings are computed based on statisticallyderived risk factors that draw from historical data from other users orother a priori data that can be used in a decision tree or equivalentprobabilistic framework.

Further, by posting object catalog and priority lists to a remote orlocal server it is possible to review data for the user by a remotelylocated expert, e.g., at a remote computer so as to permit correctionsthat could be approved and entered, after which time updates to thesystem could be made automatically or manually.

As one non-limiting example, FIG. 7 illustrates a thermal image of anenvironment relating to a public utility in the U.S. during the day, andFIG. 8 is a plot of absolute temperature difference (° C.) for threephases of a portion of the objects previously identified in the image.In particular, FIG. 7 illustrates six objects (bushings, in this case)labeled A-F, which correspond to bounding rectangles 701-706,respectively. The objects being assessed in FIG. 8 (‘A’, ‘B’, ‘C’) arebushings for a primary operating at 345 KV, while objects ‘D’, ‘E’, and‘F’ are the secondary phases. FIG. 8 then visualizes the temperaturetrends for the three phases (AB, BC, and AC) associated with theobjects.

In accordance with various embodiments, the autonomy of the system,especially for remote sites, is greatly assisted by a mobile structures,e.g., a wheeled platform that is readily transported, that enables it tobe used at multiple sites for relatively short amount of times; this ishelpful when short term surveys are required to assess a site formarginal equipment or site behavior prior to a permanent installation,or if sites are somewhat transitory, e.g., in the case of mobileelectrical substations. Likewise, the use of an extendable ortelescoping mast or support for the system 100 enables greater diversityof field of view and flexibility in configuring multi-sensordeployments. Further, using renewable energy sources, e.g., wind orsolar or equivalent, prevents the need for an electrical connection to asite, which in the case of remote electrical substations, saves cost andtime and regulatory burden. Additionally, using wireless networks havingwide area coverage, such as is available through satellite or commercialtelecommunications services e.g., LTE, enables use of the system withoutthe need for a site network connection and therefore removes the needfor site network equipment.

Automatic Configuration of a Site

In accordance with various embodiments of the invention, the site wherethe system is installed and used for safety and intrusion monitoring canbe mapped and assessed so that objects, e.g., equipment, and spaceswithin view of the system are located and labeled, after which time theycan be observed over time for thermographic behavior and occupancy byother moving objects or organisms. The system is configured for the sitein which it is located using capabilities illustrated in FIG. 1 and thesequence of operations illustrated in the flow chart of FIG. 3 .

The method (300) of FIG. 3 begins by using an available sensing device,e.g., IR camera 102, optical camera 103, or auxiliary sensor 109 todetermine fields of view (step 301), e.g., from known parameters such asfocal length, pixel size, and number of pixels, or by retrieving it froma manufacturer database or other reference for the camera. This field ofview is then used to establish the geometry of image data for eachsensor used. Image data can then be used to detect and classify objects(302) that are found an image using one of the many techniques availablefor object detection, e.g., machine learning or model-based methods orcombinations of the two, for instance. This results in a list ofobjects, for which it is then possible to constitute object properties(303), e.g., object make or model, color, emissivity, etc. Theobject-system distances are then obtained (304), using a ranging device,e.g., if included as one of the auxiliary sensors, so that each imagedobject in the site can be located accurately in three-dimensional space.The spaces, e.g., entries, exits, interiors, exteriors, ground, sky,etc., are then detected and classified (305). As it is sometimesadvantageous to have a human review the work of a computer, the objectsand spaces can then be reviewed with an operator (306) and optionallycorrected (307) before adding the objects and spaces thus classified andlocalized to an object catalog (308). In so doing, various embodimentsof the invention enable a site to be automatically configured formonitoring with the system of FIG. 1 .

In accordance with various embodiments of the present invention, and ofthe functional modules described herein may implement or more machinelearning models that undergo supervised, unsupervised, semi-supervised,or reinforcement learning and perform classification (e.g., binary ormulticlass classification), regression, clustering, dimensionalityreduction, and/or such tasks based on the acquired images.

Examples of models that may be implemented include, without limitation,artificial neural networks (ANN) (such as a recurrent neural networks(RNN) and convolutional neural network (CNN)), decision tree models(such as classification and regression trees (CART)), ensemble learningmodels (such as boosting, bootstrapped aggregation, gradient boostingmachines, and random forests), Bayesian network models (e.g., naiveBayes), principal component analysis (PCA), support vector machines(SVM), clustering models (such as K-nearest-neighbor, K-means,expectation maximization, hierarchical clustering, etc.), lineardiscriminant analysis models.

Thermographic Measurements

In accordance with various embodiments of the invention, the systemproduces estimates of physical temperature of catalogued objects, orparts thereof. Thermography requires, at minimum, an assessment oftemperature over time. Referring again to the system of FIG. 1 ,thermography is practiced using a thermal infrared camera 103. Such acamera produces images having pixel values that are proportional to theradiance of the objects in the field of view that correspond to thepixel values; imaged object radiance, for a thermal camera, e.g., oneoperating at wavelengths proximate to 10 microns, is proportional toimaged object physical temperature. Radiance is not identically equal tophysical temperature, however, as radiance captured with a thermalcamera is always only the apparent radiance, a physical observable thathas object physical temperature as one of several contributing factors.Other contributing factors include, for example, object thermalemissivity, atmospheric radiance, radiance of the imaging apparatus(i.e., lens, window, iris or stop, etc.), radiance of nearby orgeometrically related objects, wind speed, solar radiance and angle ofincidence (i.e., as implied by time of day and geographic location).Thus, in order to accurately estimate object physical temperatures (anobjective of the invention), it is necessary to remove from the thermalcamera image data the effects of contributing factors, which bydefinition do not correspond directly to object physical temperature.Traditionally, the practice of thermography with human operatorscomprises human measurement of thermal images and estimation, sometimessupported with separate measurements, of contributing factors, alongwith subsequent correction of thermal image data which yields anestimate of object physical temperature that has a known, or at leastintended, accuracy.

One goal of various embodiments of the invention is to automate thepractice of thermography. Consequently, the system of FIG. 1 serves as aproxy for a human operator who is equipped with thermal imagingapparatus and an ensemble of other measurement devices, all of which aredeployed by the human operator to effect an accurate assessment ofphysical temperature and subsequent communication of the assessment tointerested parties for objects of interest (in this case, cataloguedobjects). The practice of thermography when well performed by a human isrepetitious, complicated, and requires significant physical,mathematical, and analytical skill and care (when poorly performed thepractice is sometimes referred to as “pencil whipping”). An advantage ofthe present invention, as a means and method of thermography automation,is that electronic systems under computer control are well suited totasks that are repetitious and complicated.

The mathematical details for determining object temperatures andradiance values can be found, for example, in U.S. patent applicationSer. No. 16/779,622, filed Feb. 2, 2020, the entire contents of whichhave been incorporated by reference herein.

Image Registration

In accordance with the invention, the objects at the site where thesystem is located will be imaged recurrently so as to assess theirphysical temperature over time. In order to assure consistency ofmeasurement and given that the field of view of the camera(s) used mayvary with time, e.g., owing to unintended vibration and subsequentmotion of the support structure for the camera(s), the use of imageregistration, or pixel mapping, is contemplated with the invention. Inthe registering of images, a suitable reference image is selected for aninterval of time, e.g., manually or using predetermined imagecharacteristics to automatically select an image, and then all otherimages in that period of time are spatially adjusted so as to collocateobjects by pixel location in space and time.

There are numerous methods for automatically registering images that arewell known to those skilled in the art. In order to avoid the need forhuman intervention to produce registration results using such well knowntechniques, it is important to automatically identify image features,particularly locations where objects of interest are stationary and canreliably be used for registering temporally distinct images. A preferredembodiment for finding stationary objects uses a Datacube of thermalimagery, representing a time series of two-dimensional thermal images,as a means of identifying features to register, as follows:

-   -   1. Compute an edge representation of the data cube using spatial        filtering of each 2D image contained in the Datacube, e.g., a        Sobel or similar two-dimensional derivative based technique.    -   2. Binarize the edge representation of the Datacube.    -   3. Temporally integrate the Datacube for multiple temporal        statistics, e.g., minimum, maximum, mean and variance, and use        these to assess the spatio-temporal stability of edge features.    -   4. Identify edge features having low spatio-temporal variability        and high levels of occurrence, such features representing the        most probable stationary features.        Iterate through the edge features, selecting those with most        favorable statistics first, masking these features from further        consideration, and progressing to the next most favorable        feature, etc., until the feature list is exhausted.

Having automatically produced a reliable set of image features, one ofmany image registration algorithms can be used to map edge features in agiven image to corresponding image features in a reference image, themap thus produced permitting the calculation of corrections to apply tothe given image to enforce spatial correspondence to the referenceimage.

While the preferred embodiment of feature selection makes use of edgefeatures to identify features, it is here contemplated that othermorphological features, e.g., corners, rectangles, circles, othernon-geometric features having measurable statistics, can be used in asimilar fashion to produce sets of features that can be used to comparepairs of images and spatially register one to another.

Further, given robust and accurate image registration techniques, it iscontemplated that the invention will be used to locate objects withsub-pixel accuracy over time, which, in turn, enables the assessment ofpixel-scale temperatures that would otherwise be impossible to observereliably over time.

Corona Mapping

In accordance with one embodiment of the invention, object imagery isproduced over a time period for diverse objects of interest, includingelectrically energized objects having electric field intensitysufficient to ionize air molecules (corona discharge) in proximity tothe invention, e.g., in the air space near high voltage transformerconductors or bushings. This ionization is observable with the inventionas collections of point sources that appear as a cloud-like structure inthermal imagery. The spatio-temporal behavior of the ionization is anindicator of state for the energized apparatus and can be used to makeassessments of apparatus state that support predictive maintenance andfailure onset. The invention thus contemplates the use of thespatio-temporal behavior of imagery of ionization events, e.g.,performed using existing morphological detection and tracking algorithmsknown to those skilled in the arts of computer and machine vision, toassess the physical condition of proximate energized structures anddevices.

Image Temporal Evolution

In accordance with various embodiments of the invention, object imageryis produced over a span of time such that temporal effects in 2D and 3D,e.g., when range data is integrated with thermographic data, can beobserved. This corresponds approximately to the use of time lapse video,which when applied to thermographic data, can include analyzing timelapse video as a class of object detection and tracking, the object inthis case being a region of temperature change, e.g., hot spot or coldspot, that can propagate in an electrical circuit and the structuresassociated with it. Treating three-dimensional heat propagation as aThermal Object detection and tracking problem enables the re-applicationof many robust and mature algorithms in the domains of machine andcomputer vision. The invention contemplates the fact that differentobject classes, e.g., switches, fuses, arrestors, bushings, will havedistinct shapes of heat propagation and that, as these shapes evolveover time, they will constitute different “motions” for the thermalenergy that is propagating. As such, algorithms presently in use totrack and interpret human behavior based on motion sequences can beapplied to event detection in a thermographic setting. For instance, inthe same way that patterns of human hand motion can be interpreted asvarious signals, e.g., sign language or commonly recognized gestures,patterns of heat propagation can be interpreted as various physicalphenomena, e.g., loose connector, cracked bushing, motor bearingfatigue, etc.

Further, again viewing temporal sequences of thermal image data for ascene as a 3D Datacube, calculating a 2D image of pixel-wise temporalbehaviors, e.g., mean, variance, intra-scene correlation, frequencydomain filters, or other metrics derived from comparisons to physicalmodels, allows identification of object features of interest in thespatial domain—temporal behavior can be detected using the spatialdomain. Features of interest in this type of analysis includenon-energized surfaces, surface emissivity and air convection surfaces.Such a view of data also permits quick analysis of trends betweenobjects, such as the temperature differences between bushings for thethree phases of a distribution transformer.

Trend/Change Detection Approach

In accordance with one embodiment, multiple solutions are calculatedusing deterministic, regressive, and stochastic means, e.g., GMM(Gaussian Mixture Model) or AI/ML, autoregressive, linear time orfrequency domain techniques to estimate changes and trends, based on(e.g., diurnal) cycle-normed data. For example, the system can use theambient and peripheral information, characterized, for example, bynon-energized measurable surfaces or objects or environs. Such a processmight include:

-   -   1. Pre-computing periodic behaviors of data over time, e.g.,        diurnal cycle for the available variables (columns in the        Datacube);    -   2. Estimating current physical temperatures using current        measurements of camera, lens, window, scene, environment and        other contributing factors for which data are available;    -   3. Detrending estimated temperatures with respect to the        unenergized context (not part heat generating mechanism of the        equipment);    -   4. Normalizing current data using periodic data elements;    -   5. Weighting the normalization in accordance with physical        parameters and observed patterns for this site or environment;    -   6. Adding detrended data to a stochastic model, e.g., GMM;    -   7. Updating the regressive model, e.g., an autoregressive model;    -   8. Updating the linear model, e.g., via Fourier analysis or a        simpler approach.    -   9. Estimating trends using detrended data as a function of        multiple time periods, e.g., hourly, daily, weekly, monthly, and        calculate a likely next state for each time period, e.g., next        hour, next day, next week, next month;    -   10. Using trend and state estimates to produce warnings and        alarms;    -   11. Using linear, regressive, and/or stochastic estimates to        assess change; and    -   12. Sending changed data to a gateway for display, e.g., an        hourly, daily, weekly, monthly, change map (panorama or        composite image).

In general, the above steps are preferably performed using Datacubefunctions. Pixels in a data cube have a known relationship, but are notnecessarily adjacent or touching physically. Such an arrangement ofpixels at a point in time is panoramic in the sense that a sceneconstructed from more than one field of view is described by the datapanorama, and a Datacube is a time series of such panoramicconstructions. The known relationship between pixels spatially allowsfor their decoupling spatially, e.g., using super resolution techniques,and also for decoupling temporally, e.g., using the above steps fordetrending, which is a means of decoupling distinct data vectors, e.g.,the removal of ambient temperature effects from an object radiancemeasurement involves removing elements of the radiance that correspondto, or couple with, the ambient temperature.

Deduction and Use of Site Schematic Data

In accordance with the invention, the objects at the site image data aregathered are often related to one another as elements of an electricalcircuit. When this is the case, one can use information about theviewable objects and support structures to inform a circuit diagramaccessed from separate site design data. Given such a circuit diagram,the objects viewed and identified, e.g., manually or with computervision techniques, at the site can be associated with circuit features.The thermal data subsequently gathered for objects can be used tointerpret electrical loads using known physics, e.g., Ohm's Law, nodalanalysis, and other analytical tools known to those skilled in the artof circuit behavioral analysis. Such treatment of the site data alsoenables the use of thermal data to support so-called “digital twin”strategies, wherein sensor data gathered for a designed system are usedto update companion physical models of the system such that system statein the present and future can be estimated and exploited, e.g., for theassessments of state root causes or collateral effects.

Incorporation of Collaborative Sensors

In accordance with the invention, given connectivity permitted by acommunications network or the signal connections of the computerprocessor that is integral to the invention, a multiplicity of sensorscan be used to make assessments of site state as a function of time. Forexample, video security systems or unattended ground sensors (UGS) inproximity to the installed invention can be used to cue the inventionfor monitoring intrusions at the site. Alternately, UGS havingcalibrated thermal sensors, e.g., spot sensors, integrated into thestructure that supports the invention or located independently and inproximity to the invention, can be used to either cue the invention tothe presence of intrusion or thermal events or can be used for groundtruth that supports algorithmic techniques for constraining solutions,e.g., for object physical temperature or emissivity, such as calibrationsources having known physical temperature and emissivity. Further, asinstallation sites may often have other independent data collectionsystems the data from these may also be used by or with the invention tofocus the observations on regions of heightened interest, e.g., hotspots or locations of probable anomalies.

Use of Reference Points as Constraints

In accordance with the invention the observed behavior of thermalObjects, including with reference to independent measurements, e.g.,spot measurements with hand instruments or additional devices integratedwith the invention, e.g., UGS, will produce assessments of site regionsfor which there is elevated accuracy and reliability, e.g., objects orregions of known or having highly confident assessments of physicaltemperature and emissivity. By integrating independently collectedcomparison data for objects or deducing these from temporal behavior,e.g., permitted by Datacube analysis, it is contemplated that anchorpoints for constraining solutions can be automatically produced. Usingsuch high confidence points enables more robust solutions by addingnon-spurious information to the solution spaces. And in simple cases, itenables the automation of inter-object relative thermal trending.

Use of Scene Based Optical Characterization

In accordance with various embodiments of the invention, the objectsthat can be observed include, when they are within the field of view ofa given module, without limitation, the sun, moon, stars and other knownpoint sources or sources having well defined features in the field ofview. In order to optimize the resolution of the system, it iscontemplated that known point sources can be used to estimate theoptical performance over time as concerns resolution, e.g., the pointspread function (PSF) or equivalently the modulation transfer function(MTF). Knowing such behaviors permits improving the resolution andthereby the thermal accuracy of the system using techniques known tothose skilled in the art, e.g., deconvolution or more sophisticatedtechniques such as the CLEAN algorithm, etc. Further, by observing theoptical behavior over time, after accounting for known atmosphericvariables, e.g., water vapor content, the invention can be used todeduce the optical effects of actual atmosphere conditions along theoptical path, e.g., the blur induced by multiple scatter in the verticalatmosphere vs. the horizontal atmosphere. Finally, knowing opticalparameters for the invention and its environment supports improved imageoptimization such as super-resolution.

Object Distance Measurements

In accordance with various embodiments of the invention, the objects atthe site where the system is located may be assessed for their distancefrom the invention, such that objects can be accurately placed inthree-dimensional space, e.g., global position data or other means oflocating objects in three dimensional space, in order to make furthermeasurements of physical properties, sizes and relationships of objectsby themselves and in relation to other objects.

The system is configured for the site in which it is located usingcapabilities illustrated in the embedded system module 100 of FIG. 1 andan auxiliary sensor 109 having range finding capabilities, e.g., a laserrange finder, or a stereo pair of apertures formed by visible 102 andthermal 103 cameras or an equivalent pair formed with an imagingauxiliary sensor 109 or another embedded system module 140 in networkconfiguration with the embedded system module 100, such that range toone or more regions of interest in the field of view can be estimated,e.g., using direct measurement in the case of a laser range finder andindirectly by way of image analysis of stereo pairs using imagedisparity functions.

For ranging through the use of stereo-pair images, it is wellestablished that stereo-pair imagery can be used to deduce distancerelationships between an imaging apparatus and an object if the objectcoordinates (corresponding to physical distances in the focal plane ofthe camera being used) in the images and the physical image device focalplane separation are known. The relationship between these two distancesis described as a disparity function and is well known by those skilledin the art. In the simplest case it is described by z=f*b/d, where z isthe distance to the object, f is the focal length of the camera optics,b is the separation between images (focal planes) and d is the distancebetween objects in the stereo-pair images.

With reference to FIG. 1 , an embedded system module 100 can be combinedwith one or more such equivalent devices 140, 141, 142, e.g., identicalor functionally equivalent, such that multiple stereo pairs can beformed to produce a statistically more robust estimate of object range,and so that significant baselines can be obtained that permit enhancedrange resolution as compared with that of a single module having abaseline limited to its physical dimension. In such a case, multipleobservations can be combined to form more robust or complete range maps,e.g., regression can be used with a disparity function to compute objectrange from the slope of the multiple observation regression—the slope ofthe regression (which is b/d in the relationship z=f*b/d) beingproportional to the object range.

For example, a regression across multiple stereo observations couldproceed by using imagery from module 100 as a first image and referenceto which subsequent images will be compared, and differences computed asadditional observations (images) are added to the ensemble of imagesused to estimate range. Such additional images could be from within themodule itself, e.g., a color and thermal image pair, or a pair of colorimages from the module by virtue of an auxiliary color image sensorproviding a second color image, or by combining color or thermal imagesfrom multiple modules 100. Given known object features in each image andfeature locations (pixel coordinates) one can attempt a regression tooptimize signal to noise ratio (SNR), where SNR here is a statisticformed by the ratio of the major axis of the ellipse formed by thetwo-variable regression data scatter (as one encounters in atwo-variable scatter plot) to the minor axis of the ellipse. In thiscase we are considering “signal” to be the object feature displacementand “noise” is the scatter of signal perpendicular to the regressionline drawn through the plot of object feature displacement vs. cameradisplacement. This example is assuming a perfectly linear relationshipfor simplicity of discussion; it is contemplated that the relationshipwill be nonlinear. Imagery could be added and solutions iterated untilthe SNR is larger than a predetermined threshold.

There are many ways to improve the distance estimates. One that iscontemplated for this system 100 is the use of super-resolutiontechniques to improve the resolution of the object displacement in imagecoordinates, the super-resolution being a means of computing new,smaller equivalent pixels in a focal plane using sub-pixel angular datacollected from imaging a point source or and deducing a focal planeresponse, with subsequent de-blurring of images based on a priori knownoptical properties or optical properties of the camera lens (e.g., lenspoint spread function) measured in situ or in a laboratory as part ofthe manufacture process. In this way, the distance measurement may beimproved.

Another way to improve distance estimates is through the use of imageaveraging, or stacking, as it is sometimes known, to increase the signalto noise ratio in an image by effectively increasing the integrationtime for each pixel in the image.

Image Super-Resolution

In accordance with the invention, the objects and spaces observed withthe invention, may be observed with greater fidelity, either forthermographic or intrusion purposes, with increased image resolution,e.g., more pixels per image or more pixels per degree of optical viewingangle. A known technique for achieving this purpose is super-resolution.Generally speaking, this technique involves combining multiple images ofa scene, and subsequently combining these images so as to improve theresolution of the original image, effectively computing additional imagepixels containing new information, that information being provided byother images.

The system is configured for the site in which it is located usingcapabilities illustrated in the embedded system module 100 of FIG. 1 anda sequence of operations, e.g.,

-   -   a. Acquire one or more images from module 100 having overlapping        but distinct fields of view;    -   b. Combine images from each distinct module and sensor such that        SNR to a predetermined standard is achieved, e.g., through        averaging;    -   c. Deconcolve the point spread function of each lens associated        with each focal plane array, such that one or more pixels is        generated per original focal plane array pixel;    -   d. Map or project pixels from multiple sensors on objects in the        overlapping field of view using known orientations, fields of        view, and non-uniformities of image formation elements; and    -   e. Combine pixels on objects from multiple images and multiple        imaging devices such that spatial resolution is maximized on        objects.

In accordance with various embodiments, super-resolution can also beused for fields of view that not only overlap but for which one field ofview encompasses one or more other fields of view in part or in whole.For instance, images from a camera with one resolution and anothercamera with double the resolution but half the field of view could becombined into a single, higher resolution image of the same field ofview as the lower resolution camera, such that the highest resolutionoccurs where the fields of view overlap, e.g., near the center of theimage, the non-overlapping regions of the resultant image having itsincreased resolution derived from interpolating the original lowerresolution image. In such instances, the camera PSF can be used todeconvolve imagery prior to combining images so as to maximize theinsertion of new information into the resultant higher resolution image.

Additionally, if the camera with lower resolution and wider field ofview is radiometrically calibrated, a higher resolution calibrated imagemay be computed by combining a relatively uncalibrated high-resolutionimage with the calibrated low resolution image, e.g., including when thelow resolution device has only one pixel, using the low resolution imageas a “tie point” for the calibration of pixels in the higher resolutionimage, which for instance may be an average over a region in the case ofa single pixel or “point” sensor. In this way a more costly calibrateddevice may be used to produce enhanced imagery without the expense of alarger focal plane array.

Furthermore, if a panorama, composite, or wide area image is formed witha high resolution sensor, e.g., such as might be used once duringcommissioning of the module 100, this panorama may be used incombination with a lower resolution camera to produce higher resolutionimages at the time of subsequent measurements, by interpolating new,higher resolution pixels for combination with/into the lower resolutionand, typically, calibrated image. This approach assumes a relativelystatic background condition for the panorama, e.g., a space or equipmentassembly that does not move with time, so that the shape of the spacerepresented in the panorama can be used to produce a similar shape inthe otherwise unresolved pixels of a lower resolution image that occurswithin the image extend of the high resolution panorama. This techniquebenefits from knowing the mapping of the instantaneous field of view foreach pixel of each of the cameras (e.g., low resolution and highresolution) with respect to one another, and their PSF as a function oflocation in the focal plane array, such as can be obtained through aboresight alignment and optical characterization laboratory measurement.

Human Core Temperature Estimation

With reference to FIG. 5A and in accordance with one embodiment of theinvention, the object of interest may be a human system operator 502that is in an enterprise (or environment, such as an enclosed space)500, including but not limited to instances wherein the human 502 isproximate to equipment 504 being controlled and monitored. Operator 502utilizes thermal camera module 503, and thermographic imaging involvesthe estimation of human core temperature for assessing suitability forusing a control system 501 to operate a machine 504. Machine 504 mayalso be monitored with a thermal camera module 505 such that its healthand safety for performing the intended industrial action can be assessedand used to affect the progression of a control system 501, which is incommunication with machine 504, operator 502, thermal camera modules503, 505.

In accordance with various embodiments of the invention, thisapplication includes in its design the structures, e.g., hardware andsoftware, required to accommodate regulatory constraints, e.g., NIST,HIPAA, NERC CIP, such that the dual use of thermography for industrialasset health and safety assessments and/or industrial worker health andsafety assessments can be made and used in full compliance withgovernment regulations, e.g., the compliant combination of electronicmedical records (EMR) with industrial safety and security records anddata structures. Further, the invention contemplates the integration ofthermographic sensors with systems in industrial control, e.g.,supporting devices/systems using protocols such as OPCUA, ModBus, DNP,access control, e.g., devices/systems using protocols such as OSDP,security and operator safety such that assessments of asset and humanthermal state can be made and used throughout an enterprise.

For example, with reference to FIG. 5B, in order to reduce thelikelihood of disease propagation in an enterprise 510, thermalassessment of fever state of a worker 512 may be made using a thermalcamera module 513 as a prerequisite to gaining access to a workspace514. This access is provided by way of an access control system 511 thatprevents worker 512 from operating a piece of equipment that processesfood in a way that could serve as a disease vector mechanism. Such aworkspace may also be monitored itself with a thermal camera module 515such that the state of workers within that space may be monitored andassessed. The invention consequently further contemplates the use ofsoftware automation to create scalable computational structures, e.g.,infrastructure as code including but not limited to YAML automation forconfiguring and deploying services that integrate functions, servicesand data that span the spaces of industrial safety and security, accesscontrol, personal medical data.

An illustrative example, the following sequence provides just oneexample of how the systems and methods of the invention can be appliedto a human core temperature application using an instance of theinvention placed in a location wherein humans could obtain an estimateof their core temperature, illustrating one of many possibleembodiments:

-   -   1. Position the human in front of the system, e.g., module 100;    -   2. Capture thermal and color imagery over a brief time sequence        using spectrally optimum illumination for the human;        -   a. Optional: augment thermal data with imagery of oral            interior regions, repositioning the human accordingly, or            using multiple modules 100;        -   b. Optional: augment thermal data with in-ear data,            repositioning the human accordingly, or using multiple            modules 100;    -   3. Simultaneously measure the range (linear distance) to the        human;    -   4. Use image registration techniques to remove any human motion        artifacts from the image sequence captured and increase the        image quality for both the color and thermal imagery;    -   5. Extract human body feature locations, e.g., eyes, nose, etc.,        using color camera face recognition technology;    -   6. Use predetermined, e.g., from a bore sighting practice that        is part of manufacture and calibration, mappings between thermal        and color and knowledge of range to map body feature locations        onto thermal imagery;    -   7. Use known feature locations to extract observed thermal flux        from specific regions of the imagery gathered;    -   8. Compensate measured flux for ambient conditions including        daily temperature variation, viewing angles, using flux        corrections obtained from ongoing regression analysis;    -   9. Map flux into physical temperature using optimal estimates of        emissivity; and    -   10. Report physical (core) temperature for the human.

Thermographic Calibration

In some situations, it is advantageous to provide thermographic datahaving elevated accuracy or resolution, including but not limited tosituations where an uncalibrated thermal camera is used, a low costmarginally calibrated device is available, or when the only means ofviewing thermal data is with a video image viewing device, e.g., a colormapped screen showing a “heat map” portrayal of a camera field of view.In these situations, a variation of the invention can be configured as acalibration target having known thermographic properties, e.g.,emissivity, physical temperature, surface orientation.

FIG. 4 illustrates an embodiment in which the thermal camera module 100has been replaced in FIG. 1 by a thermographic calibration target 400,such a target 400 could further be configured with or with a localcomputer 401 such that target properties could be controlled, assessed,and communicated. The use of active calibration surfaces 403, e.g., avariable temperature or variable angle surface control enables multiplecontrolled zones or regions on the target to be used to establish acalibration curve, a calibration set point, or combinations thereof, soas to compare the calibration target directly with an object of interestby placing the target proximate to the object of interest. Passivecalibration surfaces 402, e.g., surfaces with variable emissivity, couldalso be used, though with greater limitations on the number ofvariations possible than programmable surfaces, noting that emissivityvariation is unlikely to be programmable and thus would be a passivevariation used in addition to active surfaces when such were preferred.In either passive or active calibration surface variation or acombination of the two, such a scenario enables direct comparison withdigital image data and a computer algorithm, or a visual comparison,e.g., one region represents “normal” and another “abnormal” such that ahuman operator could, using a suitable colormap, observe visually theimage regions with multiple surfaces for immediate fever screening.Also, having passive and active variation of calibration surfaces allowsfor simulating the effects of different surfaces in a scene for directcomparison with the actual object and thermal camera sensor data.Reference numerals 410-433 in FIG. 4 correspond to reference numerals110-133 in FIG. 1

Further, it is contemplated that the use of orientation sensors (408,108) for a thermographic calibration target and/or thermal camera modulewould permit calculation of relevant radiometric angles. In this way,angular dependencies of radiance can be accounted for and minimized as asource of error.

It is further contemplated that, when thermographic calibration includesthe use of static known reference temperatures, or set points, such datacan be derived using a thermographic calibration target 400 or usingaccessible scene and environment parameters, including measured, modeledor independently collected and distributed, e.g., via an internet dataservice, e.g., NOAA or NWS.

With further reference to FIG. 4 , and considering the use ofmulti-spectral sensors as Auxiliary Sensors 109 or as part of aNon-Thermal Imager 102, the calibration target may be used to emitphotons having known radiometry e.g., wavelength, radiance, etc., suchthat an Auxiliary sensor 109, Non-Thermal Imager 102, or even a ThermalInfrared Camera 103 could make use of the emission of photons as part ofa calibration routine wherein additional spectral behaviors are used toimprove the thermographic data, e.g., by adding equations to a solutionspace to reduce the number of unknowns.

Data Communications

In keeping with known practices, it is contemplated that the Device Busor Network Interface will include both wired and wireless communication.Wired communication may include but is not limited to commercial andindustrial standards including but not limited to TCP/IP, I2C, ModBus,or other means of connecting to networked computers or devices includingprogrammable logic controllers (PLCs) or other such control relateddevices. Further, when using wireless communication, commercial WiFi,Bluetooth, and variations that include mesh or other configurations arecontemplated for the invention. End points of connection contemplatedinclude remote servers e.g., cloud resources, and structures ofcommunication include regulatory standards, e.g., NIST, HIPAA, and otherknown government standards.

It is also contemplated that, in some situations, it will beadvantageous to address privacy concerns, e.g., when humans are imaged,either intentionally or unintentionally. In such a situation theinvention contemplates the storage of data on data storage elementsowned by the human for whom privacy is a concern, e.g., their phone, aportable magnetic storage device, or other such compact means ofstoring, protecting and carrying digital and personal data.

Additional Modalities

In situations where a plurality of thermal camera modules is installedin a region of observation, it is anticipated that it will be useful todetermine the interrelationship of modules with respect to one another.This is readily supported through the use of attitude sensors, e.g., toestablish spatial extents and geometries for optical axes and fields ofview, projected light or equivalent active sources, e.g., to measurefields of view and their potential overlaps, such techniques beingfamiliar to those practiced in the art.

It is also contemplated that it may be advantageous to replace oraugment the use of a thermographic calibration target 400 with emissiveor reflective surfaces that can easily be attached to surfaces in thefield of view, e.g., a “tape” type of device, such a device having knownradiometric properties and for which the physical temperature can beinterfered adequately from ambient or peripheral object temperatures.Locations of convenience would include the mullion of an entrance, forinstance, when observing humans, or a portion of a piece of industrialequipment when the context is industrial asset monitoring.

In accordance with one embodiment, the system further includes aplurality of thermal camera modules connected to the computer processorthrough its network interface.

In accordance with one embodiment, the system further includes a thermalcamera module housing design that is compatible with the presence ofhigh electric and magnetic fields by virtue of the material used, e.g.,elevated dielectric breakdown voltage, and the geometry of surfaces ofthe housing for the module, e.g., minimization of dE/dX below relevantenvironmental and material contexts, where E is the electric field and Xis a spatial variable with the “d” prefix denoting a derivative operatorfrom the calculus. Such a design enables the placement of a module 100in close proximity to energized objects, e.g., bus bars or switch gearelements, without harm to the module contents.

In accordance with one embodiment, the system further includes a cablingdesign that withstands high electric field exposure without dielectricbreakdown, or related arcing or high voltage discharge events. Such adesign could be accomplished, for instance, by a dielectric coating,e.g., in the simplest case a heat shrink tubing with dielectricbreakdown rating and thickness constituting a breakdown voltage greaterthan the anticipated electric fields of the sensing environment,encompassing and producing electric field continuity with, the sensorbody 909 its connectors 903 and associated cable 901 so as to minimizedifferential electric fields across surfaces and, when possible, producea negligibly small electric field between sensor body 909 and localground within the assembly.

In accordance with one embodiment, the system further includes, aFresnel or equivalent optical element that can be used as a thermalcamera housing lens cover and that will alter the field of view inpredetermined ways, e.g., to “steer”, enlarge, reduce or otherwisechange the field of view of the imaging sensor.

In accordance with one embodiment, the system further includes the useof multiple thermal camera modules to set up lines of site enclosingspatial cells or plurality of adjacent cells comprising a larger regionthat can be used to track objects or organisms across cells to infer atrack across the region.

In accordance with one embodiment, the system further includes the useof a calibration target with multiple zones having variabletemperatures, surface angles, material properties, e.g., emissivities,so as to span the space of possibilities for a given application, e.g.,a temperature sensitive process requiring heightened precision oraccuracy, or an observation application wherein the object of interestis a living organism, e.g., a human. Furthermore, such calibrationtarget could also be constructed by coating selected surfaces within thefield of view with a black body coating, e.g., flat black paint in thesimplest case, of known emissivity such that physical temperature of theenvironment could support estimation of surface physical temperatures.

In accordance with one embodiment, the system further includes the useof a computer controlled calibration target with network connection thatpermits connection to local, e.g., “on premise”, and wide area, e.g.,“cloud”, networks that further permit interaction with the system orother devices having access to the interfaces for the calibrationtarget.

With reference to FIG. 6 and in accordance with one embodiment thesystem further includes the use of sensors having diverse fields ofview, e.g., a narrow field sensor 601 and a wide field sensor 602 bothin view of an object of interest 605. In accordance with one embodimentand for example, the system further includes the use of auxiliarysensors 109 that have a known optical axis in relation to visiblenon-thermal imagers 102 or thermal infrared cameras 103 while alsohaving a constrained field of view, e.g., a “pencil beam”, 603 or atleast having known extents within the field of view of the wider field604 thermal infrared camera 103, and that also have enhanced calibrationfeatures such that they can be used to “anchor” thermographic image datato a high confidence. This use of additional special-function sensorsenables both an enhanced estimation of physical temperature but also, inthe event that additional spectra are measured in the field of view withincremental special-function sensors, permits estimation of additionalparameters, e.g., the presence of moisture on a surface, estimated byway of its index of refraction. Surface water content objects, e.g.,condensation on equipment or sweat produced on human skin, can also beestimated using polarimetric measurements, such as are possible withcommercial polarimetric focal plane arrays, e.g., such as manufacturedby Sony; such a polarimetric focal plane array could be used as thevisible non-thermal imager 102, for instance.

In accordance with one embodiment, the system further includes ancillarydevices for harvesting energy from fields, vibrations, motions, or heatsuch that, in a low power mode or state, the system and/or associated orproximate calibration target(s) can operate without external powersources.

In accordance with one embodiment, the system further includes a thermalcamera module configuration that does not require a computer within themodule but that relies on a local computer connected by means of anetwork interface.

In accordance with one embodiment, the system further includes a meansof connecting thermal camera modules sequentially (“daisy chain”), e.g.,using a network protocol such as CANBUS, RS422, TIA-1113, MoCA, etc.,that supports such communication. We include in this the use of WiFiover coax, hereafter WiFoX, in which a WiFi transceiver antenna port isconnected directly to the coupled port of a radio frequency tap, e.g.,Mini-Circuits RBDC-20-63, the tap being inserted serially into a lengthof coaxial cable by way of the input and output ports of the tap. ThisWiFoX connection also permits electrical power to be provided along itstwo coaxial conductors (e.g., center conductor and shield), so as tocompletely provision connected devices with power and data orcommunication. The WiFoX connection originates from a WiFi transceiveracting as a network access point or router and connects to one or moreconnected devices, with the last device connection including a matchedimpedance termination at the output port of its corresponding radiofrequency tap. Using WiFi in this way permits a wide bandwidth, secureconnection with robust tolerance for electromagnetic interference thatis cost effective by virtue of its re-use of commonly available and welltested communication protocols. Furthermore, WiFoX extends to mixedwired and wireless communications trivially by way of inserting anantenna, e.g., with amplification if needed, into the daisy chainarrangement in place of a system module.

In accordance with one embodiment, the system further includes a meansof sending image data over the network connection, e.g., CANBUS, RS422,TIA-1113, MoCA, WiFoX, etc., by packetizing the image data and sendingincrements of a partial image so as to support small packet size formatsconsistent with protocols having small packet sizes (CANBUS beingexemplary in this regard). Further, in order to minimize transmissionbandwidth used on a network when many devices are daisy-chained andcapable of sending data in temporally small intervals, the means ofsending image data includes sending partial images of one or morepicture elements (pixels), wherein such pixels are selected fortransmission because their state has changed. This state change, in sucha case, is assessed by various means, including but limited to, simplemethods such as the magnitude of the mathematical difference betweensuccessive or time-filtered measurements of the pixel, or morecomplicated methods in which the pixel state change is assessedstatistically by well-known means, e.g., Gaussian Mixture Models, inwhich a decision is made as to whether the pixel is in a backgrounddistribution or a foreground distribution, including when multiplepredetermined foreground distributions are considered.

In accordance with one embodiment, the system further includes a meansof connecting thermal camera modules 100 or calibration targets 400 as awireless network, e.g., mesh, of devices. In accordance with oneembodiment, the system further includes a means of connecting thermalcamera modules in a star network configuration.

In accordance with one embodiment, the computer processor is configuredto perform a self-configuration procedure based on objects detected andclassified at the site during set-up, substantially without humanintervention.

In accordance with one embodiment, the power source is a renewableautonomous power source drawn from the environment at the site, e.g.,solar, thermal, vibrational, electric field, magnetic field, or othersuch “energy harvesting” means.

In accordance with one embodiment, the computer processor is configuredto perform the detection and classification of objects of interest, orcomponents or elements of objects of interest, e.g., connection pointsin an electrical buss connection, using at least one machine learningmodel.

In accordance with one embodiment, the computer processor is furtherconfigured to perform intrusion detection based on the plurality ofthermal images and send an alarm and/or an attendant additionalinformation, e.g., thermal image, color image, environmental data, etc.,via the network interface when such an intrusion is detected.

In accordance with one embodiment, the system further includes at leastone auxiliary GPS sensor configured to sense the location of the thermalimaging system and utilize that location data in producing the statedata.

In accordance with one embodiment, the computer processor is furtherconfigured to use a Datacube time-series data structure for determiningthe state data. This computer processor configuration includes, by wayof example, use of the Datacube data to detect anomalies, e.g., usingmachine learning techniques for exploring such multi-dimensional data.Such detection can further be used to initiate additional assessments,e.g., computational or human or both, of spatio-temporal trends in thedata, including when data are assessed jointly or serially with otherdata collected, e.g., environmental data or measurements related to thenearby equipment and its environment.

In accordance with one embodiment, the computer processor is furtherconfigured to estimate corona effects, including partial dischargephenomena, for a high-voltage object of interest, using either thermalinfrared sensor data or radio frequency emissions, such radio frequency(RF) emissions being measured with RF receiver devices having a passbandat appropriate frequencies, e.g., proximate to 100 MHz or other knownpassbands for such phenomena, such measurements assisted by knownsoftware defined radio (SDR) techniques such as those produced by Ahmed,et al. (2019). Partial Discharge Detection and Localization: UsingSoftware-Defined Radio. IEEE Industrial Electronics Magazine. 13.10.1109/MIE.2019.2942209. Alternately, an existing RF receiver, such asis used in included communications devices, e.g., WiFi or Bluetooth, canbe used to monitor the background noise levels by virtue of the noiseterm implicit in a measurement of SNR, or signal to noise ratio, such asis commonly available from RF transceiver devices. In this way boththermographic and electromagnetic phenomena can be used to singly orjointly to estimate the presence of connection, insulation, etc.,anomalies in equipment being monitored. Further, the data resulting frommeasurements can be used in a machine learning (ML) model, includingcompact mappings of ML models, e.g., TinyML, to produce anomalyestimates in remote place (“at the edge”) with a minimum ofcomputational and therefore electrical power.

In accordance with one embodiment, the computer processor is furtherconfigured to perform a resolution enhancing process on the acquiredthermal images, such enhancement process often involving the calculationof point spread functions (PSFs) or modulation transfer functions (MTFs)from sensor data and, when possible, objects in the field of view. SuchPSF data can further be used to estimate the presence of obscurants,e.g., dust or other light scatterers, such that radiometric data can becorrected or flagged for presence of obscurants. Further, whenobscurants are probable based on PSF or equivalently derivedmeasurements, including assessments made on joint statistics of multiplephysical observables, e.g., visual, thermal imagery, humidity data, thecomputer processor is configured to activate a dust mitigationmechanism, e.g., a compact compressed air device proximate to opticalwindows or other sensing surfaces, such activation removing the dustfrom surfaces of concern.

In accordance with one embodiment, the field of view of the ThermalInfrared Camera 103 or Non-Thermal Imager 102 are aided with customoptics placed proximate to the respective focal plane arrays and/or inthe field of view so as to collect and image reflections or emissionsfrom objects that are not in the line of site. For instance a anamorphiclens could be used to stretch the field of view along one axis toenhance resolution along that axis, e.g., in a 4:1 ratio, and one ormore reflective elements placed at an angle, e.g., 45 degrees, could beplaced in the field of view that reflect emission from optics that areoccluded by objects that are in the field of view but that are notoccluded when one traces rays of light from the occluded object throughthe reflective element and into the lens of the camera which thenfocuses the light rays on its focal plane array. In such a way opticscould be used to enhance fields of view or reach obscured areas that arecommonly found in restricted areas, e.g., switchgear assemblies incabinets.

In summary, an automated thermal imaging system in accordance with oneembodiment includes: a thermal infrared camera module configured toproduce thermal images of objects at a site within its field of view; apower source; and a computer processor communicatively coupled to thethermal infrared camera module, the network interface, and the powersource. The computer processor is configured to detect and classify aset of objects of interest within image data from the system, producestate data characterizing the temperatures of the objects of interest,and transmit the state data to a remote server via the networkinterface.

Embodiments of the present disclosure may be described herein in termsof functional and/or logical block components and various processingsteps. It should be appreciated that such block components may berealized by any number of hardware, software, and/or firmware componentsconfigured to perform the specified functions. For example, anembodiment of the present disclosure may employ various integratedcircuit components, e.g., memory elements, digital signal processingelements, logic elements, look-up tables, or the like, which may carryout a variety of functions under the control of one or moremicroprocessors or other control devices.

In addition, those skilled in the art will appreciate that embodimentsof the present disclosure may be practiced in conjunction with anynumber of systems, and that the systems described herein are merelyexemplary embodiments of the present disclosure. Further, the connectinglines shown in the various figures contained herein are intended torepresent example functional relationships and/or physical couplingsbetween the various elements. It should be noted that many alternativeor additional functional relationships or physical connections may bepresent in an embodiment of the present disclosure.

As used herein, the terms “module” or “controller” refer to anyhardware, software, firmware, electronic control component, processinglogic, and/or processor device, individually or in any combination,including without limitation: application specific integrated circuits(ASICs), field-programmable gate-arrays (FPGAs), dedicated neuralnetwork devices (e.g., Google Tensor Processing Units), electroniccircuits, processors (shared, dedicated, or group) configured to executeone or more software or firmware programs, a combinational logiccircuit, and/or other suitable components that provide the describedfunctionality.

As used herein, the word “exemplary” means “serving as an example,instance, or illustration.” Any implementation described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other implementations, nor is it intended to beconstrued as a model that must be literally duplicated.

While several illustrative embodiments of the invention have been shownand described, numerous variations and alternate embodiments will occurto those skilled in the art. Such variations and alternate embodimentsare contemplated and can be made without departing from the spirit andscope of the invention as defined in the appended claims.

The invention claimed is:
 1. A thermal imaging system comprising: a base unit including a first processor, a first cable interface, and a network interface; a plurality of sensor modules, each sensor module including: (a) at least one thermal infrared camera configured to acquire thermal images of objects in an environment within its field of view, (b) a second cable interface and (c) a third cable interface; wherein the first, second, and third cable interfaces of each sensor module are configured to accept a cable of a type that provides both data communication and power, and the sensor modules are configured to interface with one or more additional sensor modules arranged in series; wherein each sensor module is configured to transmit the thermal images to the base unit, which is configured to produce a composite image including a set of the thermal images received from the sensor modules; and wherein the system is configured to perform a self-configuration procedure based on objects detected by the sensor modules, substantially without human intervention.
 2. The thermal imaging system of claim 1, wherein the sensor modules are configured to be mounted within a switchgear cabinet to acquire thermal images of objects located therein.
 3. The thermal imaging system of claim 1, wherein each sensor module further includes an optical camera.
 4. The thermal imaging system of claim 1, wherein each sensor module further includes an illumination source.
 5. The thermal imaging system of claim 1, wherein the sensor modules and base unit are configured to communicate in a daisy chain topology.
 6. The thermal imaging system of claim 1, further including a mobile platform configured to change the position and orientation of the sensor modules relative to the environment.
 7. The thermal imaging system of claim 1, wherein each sensor module is configured to perform intrusion detection based on the plurality of thermal images and send an alarm via the network interface when such an intrusion is detected.
 8. The thermal imaging system of claim 1, wherein each sensor module further includes at least one auxiliary GPS sensor.
 9. The thermal imaging system of claim 1, wherein each sensor module includes a housing that is substantially cylindrical and is collinear with the axes of the cable interfaces.
 10. The thermal imaging system of claim 1, wherein each sensor module is configured to estimate corona effects for a high-voltage object of interest.
 11. The thermal imaging system of claim 1, wherein each sensor module is further configured to perform a resolution enhancing process on the acquired thermal images.
 12. A method for monitoring the thermal state of an environment, comprising: providing a base unit including a first processor, a first cable interface, and a network interface; mounting, within the environment, a plurality of sensor modules, each sensor module including: (a) at least one thermal infrared camera configured to acquire thermal images of objects in an environment within its field of view, (b) a second cable interface and (c) a third cable interface; wherein each sensor module is configured to transmit the thermal images to the base unit, which is configured to produce a composite image including a set of the thermal images received from the sensor modules, wherein the first, second, and third cable interfaces of each sensor module are configured to accept a cable of a type that provides both data communication and power, and the sensor modules are configured to interface with one or more additional sensor modules arranged in series; connecting a cable between the base unit and at least one of the sensor modules; and performing a self-configuration procedure based on objects detected by the sensor modules, substantially without human intervention.
 13. The method of claim 12, wherein the sensor modules are mounted within a switchgear cabinet to acquire thermal images of objects located therein.
 14. The method of claim 12, wherein each sensor module further includes an optical camera.
 15. The method of claim 12, wherein each sensor module further includes an illumination source.
 16. The method of claim 12, wherein the sensor modules and base unit are configured to communicate via a CAN bus protocol.
 17. The method of claim 12, further including providing a mobile platform configured to change the position and orientation of the sensor modules relative to the environment.
 18. The method of claim 12, wherein each sensor module includes a housing that is substantially cylindrical and is collinear with the axes of the cable interfaces. 